[CONTP-1758] Improve DatadogGenericResource reconciliation at scale#3143
Open
tbavelier wants to merge 12 commits into
Open
[CONTP-1758] Improve DatadogGenericResource reconciliation at scale#3143tbavelier wants to merge 12 commits into
tbavelier wants to merge 12 commits into
Conversation
🛑 Gate Violations
ℹ️ Info🎯 Code Coverage (details) Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: f2af21b | Docs | Datadog PR Page | Give us feedback! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3143 +/- ##
==========================================
+ Coverage 43.79% 44.73% +0.93%
==========================================
Files 375 377 +2
Lines 30575 31508 +933
==========================================
+ Hits 13390 14094 +704
- Misses 16276 16489 +213
- Partials 909 925 +16
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 13 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
tbavelier
force-pushed
the
tbavelier/thousands-of-ddgrs
branch
from
d08e2b9 to
b6051a5
Compare
tbavelier
force-pushed
the
tbavelier/thousands-of-ddgrs
branch
from
b6051a5 to
29c92c9
Compare
tbavelier
changed the title
tbavelier
force-pushed
the
tbavelier/thousands-of-ddgrs
branch
from
d4a9c5c to
136adf5
Compare
2 tasks
drichards-87
approved these changes
Jun 16, 2026
drichards-87
left a comment
drichards-87
left a comment
Contributor
There was a problem hiding this comment.
Left some suggestions from Docs and approved the PR.
Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com>
Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com>
Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com>
Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com>
Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com>
Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Improves
DatadogGenericResourcereconciliation behavior at high resource counts:DatadogGenericResourcecontroller concurrency with--datadogGenericResourceMaxConcurrentReconciles--datadogGenericResourceRequeuePeriodandDD_GENERIC_RESOURCE_REQUEUE_PERIODMotivation
At high CR counts, periodic status polling can dominate the controller queue and delay user-facing create/update/delete operations. This keeps DDGR status polling tunable and lower priority while leaving normal reconciliation and backend error retries at regular priority.
Should help in situations like #2816
Additional Notes
The low-priority path is limited to DDGR
refreshStatereconciles, which currently applies to resource types that fetch backend state such as monitors and SLOs. It intentionally does not lower the priority of create/update/delete retriesValidated manually that
workqueue_depth{controller="DatadogGenericResource", priority="-100"}exposes low-priority queue depth for status polling requeues. Created 1k DDGR with 10 concurrent reconciles, then changed to 1 concurrent reconcile to accumulate backlog in low prio queue for status update. Created an additional DDGR and verified it was treated in priority over the backlog (synced with backend)Minimum Agent Versions
No minimum Agent or Cluster Agent version changes.
Describe your test plan
2 things to test "separately":
--datadogGenericResourceMaxConcurrentReconciles=10datadogGenericResourceMaxConcurrentReconcilesback to 1 (can remove it since 1 is the default) and wait for a minute or two while the new operator pod acquires lease (or delete lease directly + restart the pod after)k apply -f examples/datadoggenericresource/dashboard-sample.yamlk delete ddgr ddgr-dashboard-sample: once again, ensure it is indeed deleted--datadogGenericResourceMaxConcurrentReconciles=10and clean up all the resources (again with a script/something)DD_GENERIC_RESOURCE_REQUEUE_PERIODenv var set to5mfoo){"level":"ERROR","ts":"2026-06-16T09:59:18.652Z","logger":"controllers.DatadogGenericResource","msg":"Invalid value for generic resource requeue period. Defaulting to 60 seconds.","error":"time: invalid duration \"foo\"","stacktrace":"github.com/DataDog/datadog-operator/internal/controller/datadoggenericresource.requeuePeriodFromEnv\n\t/workspace/internal/controller/datadoggenericresource/controller.go:91\ngithub.com/DataDog/datadog-operator/internal/controller/datadoggenericresource.requeuePeriod\n\t/workspace/internal/controller/datadoggenericresource/controller.go:82\ngithub.com/DataDog/datadog-operator/internal/controller/datadoggenericresource.NewReconciler\n\t/workspace/internal/controller/datadoggenericresource/controller.go:69\ngithub.com/DataDog/datadog-operator/internal/controller.(*DatadogGenericResourceReconciler).SetupWithManager\n\t/workspace/internal/controller/datadoggenericresource_controller.go:52\ngithub.com/DataDog/datadog-operator/internal/controller.startDatadogGenericResource\n\t/workspace/internal/controller/setup.go:234\ngithub.com/DataDog/datadog-operator/internal/controller.SetupControllers\n\t/workspace/internal/controller/setup.go:100\nmain.run\n\t/workspace/cmd/main.go:413\nmain.main\n\t/workspace/cmd/main.go:223\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285"}Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel